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Abstract — We investigate the problem of designing delay-aware 
joint flow control, routing, and scheduling algorithms in general 
multi-hop networks for maximizing network utilization. Since 
the end-to-end delay performance has a complex dependence 
on the high-order statistics of cross-layer algorithms, earlier 
optimization-based design methodologies that optimize the long 
term network utilization are not immediately well-suited for 
delay-aware design. This motivates us in this work to develop 
a novel design framework and alternative methods that take 
advantage of several unexploited design choices in the routing 
and the scheduling strategy spaces. In particular, we reveal and 
exploit a crucial characteristic of back pressure-type controllers 
that enables us to develop a novel link rate allocation strategy 
that not only optimizes long-term network utilization, but also 
yields loop free multi-path routes between each source-destination 
pair. Moreover, we propose a regulated scheduling strategy, based 
on a token-based service discipline, for shaping the per-hop 
delay distribution to obtain highly desirable end-to-end delay 
performance. We establish that our joint flow control, routing, 
and scheduling algorithm achieves loop-free routes and optimal 
network utilization. Our extensive numerical studies support our 
theoretical results, and further show that our joint design leads 
to substantial end-to-end delay performance improvements in 
multi-hop networks compared to earlier solutions. 



I. Introduction 

Communication networks are expected to serve a variety 
of essential applications that demand high long-term through- 
put and low end-to-end delay. Over the last decade, we 
have witnessed the development of increasingly sophisticated 
optimization and control techniques targeting flow control, 
routing and scheduling components to address the cross-layer 
resource allocation problems for communication networks. 
Among various different developed policies, an important class 
of throughput-optimal policies has evolved from the seminal 
work 1 1 1 of Tassiulas and Ephremides, where they proposed 
the well-known back-pressure scheduling/routing policy. This 
policy utilizes properly maintained queue-length information 
to dynamically determine scheduling and routing decisions 
that optimize the long-term maximal throughput levels be- 
tween all source-destination pairs. More recent works (e.g. 
El, El, m, iH; also see (61, Q, El and references therein) 
extended this framework by developing an optimization-based 
design methodology for the development of joint flow control, 
routing, and scheduling algorithms to maximize the long-term 
utilization of the network resources, measured through proper 
functions of throughput. 
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However, existing works have dominantly concentrated only 
on the long-term performance metrics of throughput or utiliza- 
tion, while ignoring the end-to-end delay metric that is crucial 
to many essential applications. This restriction allowed for 
the formulation of the design problem as a static optimiza- 
tion problem in terms of the mean behavior of the network 
algorithm operation. In particular, long-term design objectives 
can easily be described in terms of the mean flow rates and 
the mean link rates that the network algorithm provides to 
the applications. Unfortunately, this approach is no longer 
applicable when end-to-end delay is taken into account as end- 
to-end delay is a complex stochastic process that depends on 
the higher order statistics of the network algorithm operation. 
This calls for a more sophisticated delay-aware cross-layer 
design framework, and novel strategies that provide favorable 
end-to-end delay performance, while preserving long-term 
optimality characteristics. 

With this motivation, in this paper, we are interested in 
the well-founded design of cross-layer network algorithms 
for general multi-hop networks that not only utilizes the 
network resources for long-term throughput optimality, but 
also exhibits desirable end-to-end delay characteristics. Our 
contributions in this direction can be summarized as follows: 

• We propose a novel design paradigm that decouples 
the objectives of long-term utility maximization from the 
end-to-end delay-aware resource allocation. This framework 
is expected to enable the systematic development of future 
schemes, in addition to the one we develop in this paper. 

• We reveal the solution given by the back-pressure policy 
has a unique structure which facilitate us to establish loop- 
free multi-path routes that guarantee long-term network utility 
maximization. This new routing- scheme not only inherits the 
adaptive and optimal nature of the long-term optimal algo- 
rithms, but also eliminates the unnecessary loops to reduce 
the end-to-end delay without sacrificing from throughput. 

• We combine this loop-free route construction strategy with 
a token-based scheduling discipline that regulates the higher- 
order statistics of service processes to achieve drastic reduc- 
tions in the end-to-end delay performance, while guaranteeing 
long-term optimality characteristics. 

The above points also constitute the main differences of this 
work from the recent efforts in the design of algorithms with 
low end-to-end delay performance (e.g. |9|, |10|). We also 
note that there has been recent interest in deriving fundamental 
bounds on the delay performance f lTTl . |tl2|, |13|). In our 
future work, we are interested in utilizing and extending these 
results to study the gap between these fundamental bounds 
and the delay performance of our algorithms. 
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The remainder of the paper is organized as follows. Sec- 
tion [n] introduces our system model and objectives together 
with the description of several existing algorithms. In Sec- 
tion [nij we propose our delay-aware design framework that 
describes the desired characteristics and interconnections of 
the routing, scheduling, and flow control components of the 



network algorithm. In Section |IV| we build upon the proposed 
design framework to construct a delay-aware cross-layer algo- 
rithm that performs the desired tasks. The numerical result 
of the policy and our concluding remarks are provided in 
Section |V| and |Vlj respectively. 



II. System Model and Objective 

In this work, we study wired networks to concentrate 
on the delay behavior of network algorithms without the 
additional complications of interference limited wireless com- 
munications. Discussion of possible extension of our proposed 
frameworks to interference limited networks is provided in 
the conclusions. We consider a fixed multi-hop network rep- 
resented by graph Q = (A/^, c), where A/" is the set of nodes, 
C is the set of bidirectional links (i, j) where z, j G A/". We 
use c = {cij)^i^ij^^jc} to denote the vector of bidirectional link 
capacities in packets per slot, i.e., both Cij and cji refer to the 
bidirectional link capacity of the bidirectional link (i^j). For 
clarity, we use (i, j) to denote the directed link from node i 
to node j. Time is slotted in our system, and external packets 
arrive at the beginning of each time slot. 

The network resources are to be shared by a set of 
commodities. We distinguish different commodities by their 
destinations. We define V to be the set of all destination 
nodes. In this paper, we are interested in designing joint flow- 
control, scheduling, and routing policies with desirable long- 
term throughput and short-term delay characteristics. Our dual 
goal will be discussed in further detail after we introduce some 
notations. 

In each time slot, the service on the link (i, j) of commodity 
d is denoted by Rfj[t], which is assumed to be a stationary 
ergodic stochastic process. It is determined by the scheduling 
and routing policy. We let rfj = lim^^oo ] Yll=o^i^ijb']) 
to be the link rate of commodity d on link j), and define 
r = {rfj)ij^d to be the vector of all such link rates. Under 
flow control mechanism, the number of the exogenous packets 
that arrives at node s destined to node d at time slot t is 
denoted by Xf[t], which is also assumed to be stationary 
and ergodic. Similarly, = lim^^oo ^ Y1t=o ^i^sb']) the 
corresponding rate, and we let x = {Xg)s4 to be exogenous 
arrival rate vector. A utility function UsdiXg) is associated with 
each source-destination pair (s^d). We make the following 
typical assumption on the utility function: 

Assumption 1: The utility functions {Usd{xi)}s4 are 
strictly concave, twice differentiable and increasing functions 

A. Objective 

Our goal is to develop a joint flow-control, scheduling, and 
routing algorithm that not only optimizes long-term network 

^This is not a critical assumption, but will make our analysis clear. 



utilization, but also provides desirable delay characteristics. 
We discuss these two goals next. 

Utility Maximization: For the network (A/", >C,c), the net- 
work utilization maximization optimization problem is defined 
as: 



max 



s,d 
-dr. 



s.t. X^[t] > 0, V5, d G A/", Vt > 0, 

Rfj[t]>o, y{ij)ec, vt>o, 
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We define the feasible solution as: 

Definition 1: (Feasible solution) A solution {X[t]^R[t]) is 
feasible if it satisfies conditions ([2]) to ([5]). o 

We use {X* [t] , i^* [t] } to denote the optimal solution to ([T]), 
and we define 



1 '"^ 
lim -yX*[r], 

r=0 
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Note that under Assumption [T] ic* is unique while r* generally 
belongs to a set of optimal link rates denoted by 7^*. 
In this paper, we define the stability as follows: 
Definition 2: (Network stability) We say the network is sta- 
ble in the mean sense if for any queue (price) P^^ , Vn, j G A/", 



\Jd e A/-, we have limsup,^^ } EUo^ <-M 



< oo. o 



Delay Improvement: A step beyond just solving the above 
optimization, our second goal is to develop a new mechanism 
that reduces the end-to-end delay experienced by the traffic 
while maintains the utility maximizing nature. The end-to-end 
delay experienced by one packet is defined as the difference 
between the time instance of injection at the source and 
reception at the destination, which is a short-term metric 
instead of a long-term throughput. We also aim to find a new 
architecture that can decouple the different objectives. 

B. Background 

The back-pressure algorithm maintains a queue for each 
commodity d in each node i whose length at time t is denoted 
by Pf[t]. The algorithm is proposed for a general network 
model, and specifically, the back-pressure algorithm under our 
system model is given by 

The Discrete Time Back-Pressure (DTBP) Policy (d, 

EI) 

• Queue (Price) Evolution: 

Each queue evolves as 



pflt+i] 



Pflt] ■ 
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• Rate control: 

In time slot t, the flow controller determines the mean of 
the injection as 

E{Xf[t] I P^[t]) = min{ul^\P^[t]/K),X^a,c}, 

where X^ax is some finite upper bound for the arrival 
rate. 

• Scheduling and Routing: 

Each link / = (i, j) selects 

d^t] = argmax \Pf[t] - P/M| , / G (9) 

d 

and assign 

pdn^( c,j, ifd = d;[t]mdPf[t]-Pf[t]>0; .^Q. 
U I- J \ 0) otherwise. ^ ^ 

In the event of multiple commodities satisfying Equa- 
tion (|9]), we arbitrarily choose one such commodity with 
equal probability as a tie -breaking rule. 

The constant K in the algorithm is a design parameter that 
determines how close the algorithm can converge to its optimal 
solution (see |14|, |5|). Also we shall see in Section |IV] this 
solution is related to dual decomposition. 

Remark 1: In our model, since there is no interference 
between links, the back-pressure algorithm reduces to making 
decisions on each link independently. Since each link can only 
transmit in one direction at a given time slot, it is necessary 
to use the weight defined in Equation ([9]) to determine which 
one of the directional links should transmit. 



Remark 2: Note that the rate assignment in Equation (10) 
implies that we only allow transmission on those links with 
strictly positive maximum back-pressure. This does not affect 
the optimality of the DTBP algorithm. 

Remark 3: The price evolution in Equation ^ implies that 
in the case of a node has less packets than scheduled service, 
dummy packets are transmitted. Thus the allocated service 
Rfj[t] always equals to number of packets transmitted over 
link (i, j) at time t as suggested in Equation ([s]). 

There are many follow up works of the back-pressure algo- 
rithm. In particular, in ifTOl , an interesting min-resource algo- 
rithm is proposed based on the back-pressure algorithm. In this 
implementation, instead of using the queue-length difference 
I [t] - Pf [t] I as the weight of a Hnk, ( | P^ [t] - Pf [t] \ - M) 
is used, where M is some positive constant. This modification 
discourages the use of links unless the queue-length differ- 
ences are sufficiently high, and hence reduces possible loops 
in the network. 

III. Delay- Aware Design Framework 

In this section, we expose the delay deficiencies of long- 
term utility maximizing designs such as DTBP and its vari- 
ants, both conceptually and through numerical studies. In 
particular, we reveal that both the multi-path routing and 
the scheduling components of the earlier designs must be 
significantly changed or enhanced to obtain favorable end- 
to-end delay performance. Based on the observations, we 
propose a general delay-aware design framework whereby the 
utility maximization is combined with delay-aware routing and 
scheduling components to achieve our dual objective. 



A. Deficiencies in multi-path routing 

Long-term performance optimizing algorithms share the 
common characteristic of continuously searching for routes to 
maximize the end-to-end throughput of the flows. However, 
this potentially causes loops and unnecessarily long routes for 
a large subset of the packets, as we shall demonstrate in the 
following example. 

Consider a 6 x 6 grid with unit capacity links serving two 
commodities as shown in Fig. [T] The source-destination node 
pair for Commodity 1 and Commodity 2 are marked with 
square and circle respectively. Under the operation of DTBP, 
the hop-count distribution of Commodity 1 packets converges 
to the plotted distribution. We can observe that the hop count 
exceeds the maximum possible loop-free path length of 35 
for a non-negligible fraction of the packets. It turns out that 
such behavior is typical for similar long-term optimal policies 
under different setups. In this work, we are interested in 
preserving the long-term optimality of such solutions, but also 
eliminating loops and hence significantly improving the delay 
performance. We will explore the diversity in the set of optimal 
link rates 7^* to achieve this. 




Number of Hops of Commodity 1 

Fig. 1. The distribution of hop-count for flow 1 packets indicates the presence 
of unnecessary loops in their routes 



B. Deficiencies in scheduling and fiow control 

Long-term utility maximizing policies such as DTBP con- 
verge to a set of mean flow and link rates that solves ([T]). Yet, 
the end-to-end delay of these policies is a complex function 
of higher order moments of the packet arrival and service 
processes that are determined by the flow controller and the 
scheduler. 

As a simple example, it is well-known ( ifTSll ) that in a G/G/1 
queueing system, the mean waiting time W is bounded by 



W < 



(11) 



where is the variance of inter-arrival time, is the variance 
of service time, i is the mean of inter-arrival time, and p 
is the utility factor. This formulation suggests that reducing 
the variance of the arrival and service processes is crucial in 
reducing the delay. 

In a large scale multi-hop network, it is not possible to 
formulate the end-to-end delay as a function of the arrival 
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and service processes. However, motivated by the above ob- 
servation, in this work, we are interested in regulating the flow 
controUer and scheduHng components of the long-term optimal 
design to improve delay performance. 

C. Proposed multi-layer design framework 

The previous two subsections expose deficiencies in the 
routing, scheduling, and flow control components of existing 
long-term utility maximizing algorithms, and reveal new op- 
portunities in the design of delay-aware cross layer algorithms. 
These motivate us in this section, to propose a novel multi- 
layer design framework for multi-purpose policy design. 



Long Term Utility Maximization 



Optimal 
Flow 
Rates 



An Optimal 
Link Rate Vector 



Delay- Aware Routing 



Enhanced 
Link Rate Vector 



Delay-Aware Scheduling 



Fig. 2. Multi-Layer Design Framework 

The qualitative structure of our proposed framework is 
shown in Fig.[2j where the goals of long-term optimality, loop- 
free routing, and regulated flow control and scheduling are 
decoupled into different layers of the conceptual framework. 

In the first layer, we solve the problem of utility maximiza- 
tion. Note that under Assumption [T] the optimal solution for 
flow rates, is unique, while r* may have multiple solutions. 
The delay-aware routing layer exploits the freedom amongst 
these link-rate solutions and obtains a set of more desirable 
link rates that improves delay performance while preserving 
the utility maximization. Acquiring the utility maximization 
flow rates form the first layer and the delay-aware link-rates 
form the second layer, the delay-aware scheduler regulates 
the arrival and service process to achieve better delay per- 
formance. As such, this establishes a design framework for 
developing delay-aware cross-layer algorithms that possess the 
above characteristics and hence is expected to be useful for 
algorithm designs for diverse network scenarios. In particular, 
we shall next present specific algorithms with attractive delay 
characteristics. 

IV. Delay- AWARE routing and scheduling 

In this section, we present our delay-aware routing and 
delay-aware scheduling algorithms for our proposed frame- 
work. Our delay-aware routing algorithm preserves utility 
maximizing properties while avoiding loops. Our delay-aware 
scheduling algorithm implements a network stabilizing token- 
based scheme that significantly reduces queue lengths for any 
feasible rate assignment. We then combine these two schemes 
in our delay-aware, cross layer congestion control, routing 
and scheduling policy following our proposed architecture of 
Section HITCl 



A. Delay-aware route construction 

To overcome the disadvantages observed in the back- 
pressure algorithm, a natural idea is to restrict the direction 
in which packets can be transmitted over a link for a certain 
commodity. To achieve this, we first introduce a fluid approx- 
imation associated with ([T]) for the following discussion. 



max y^Usdix'l) 



(12) 



s.t. < > 0, Vs, d G A/", (13) 
4>0,V(i,j)eA (14) 

E ''i + E ^ ^^i' ^(^'^■) e >C, d e P, (15) 



' m% — / J 



{m,i)eC {ij)ec 

yieJ\f,ydeV,ij^ d. 



(16) 



Note that if an optimal solution of ([T]) is it 
is shown in previous works (e.g. lfT4ll . Tsl) that the associated 
ic*, r*) defined in ^ and ^ is an optimal solution of ( p^ 



An Alternate Optimal Solution: To ensure the optimality 
of the proposed algorithm, we explore the solution space for 
an alternate optimal solution with a particular structure. First, 
we define the following mapping of the link rates: 

Definition 3: (Delay-aware link rate mapping) The delay- 
aware link rate mapping Rfj[t] on the directed link (i,jf) for 
commodity d at time t is defined as 



RfM = 




(17) 



where d e (ij) G C and (Z)+ = max(0,Z). Note that 
this measures the running average of the net rate of commodity 
d traffic traversing link (i, j). o 



Suppose we use RfM = R*^f[t] in (17). Let 



limt^oo Rf-[t] and f* = {r*j)ij^d then is related to r* 



as: 



ij 



= { 



ij ' ji ) ■ 

_ ^*d\'^ 
ji ' ij ) • 



(18) 
(19) 



We shall show that the link rate mapping ([18]) and ([19]) 
preserves flow rate optimality. 

Proposition 1: If {x*^r*) is an optimal solution to (12), 
then (x*,^*) with = x* and f* being as defined in 
Equations ( [TS] ) and ( [T9| ) is also an optimal solution. 

Proof: The detailed proof is provided in Appendix [A| ■ 

Remark: Note that under this link rate assignment, for each 
(i, j) G £, at least one of r^f and r'-f equals to zero as if 
the corresponding bidirectional link becomes a unidirectional 
link. Clearly, those links with equal rates on both directions 
will have a net rate of 0. Also, note that this link rate 

^More precisely, (a3*,r*) is within 0{1/K) of the optimal s olut ion of 
12) . Nevertheless, we still call (a5*,r*) an optimal solution of {12) since 



tne converge can be precise if a diminishing step-size (in our notation, if K 
grows linearly with time t) is used, 
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mapping preserves the optimality of (aj*,f*) when appHed 
to any optimal solution (ic*,r*) to problem ([T]) which is not 
necessarily given by the DTBP algorithm. 

The Steady-State Behavior of the DTBP Algorithm: 
Previous works (e.g. 1 14|) also show the stability of the DTBP 
algorithm. In other words, the Markov chain with the queue- 
lengths P = {Pf)i^d being its states is positive recurrent and 
ergodic under the DTBP algorithm. We use 7r(P) to denote 
the steady state distribution under the DTBP algorithm, i.e., 
the probability of the queue-lengths being P is 7r(P) after 
the convergence of the DTBP algorithm and Xlp7r(P) = 1. 
Also, for queue Pf, the following holds: 



lim iy^^D'' 



E{pf[T]) = ypMp), 



p 



(20) 



and we use 



(21) 



to denote the optimal average queue-length of queue Pf. 



Intuitively, the link rate assignment equation ( ^01 ) of the 
DTBP algorithm implies that if there is a positive net flow 
rate from node i to node j for destination d, then the average 
queue-lengths should be strictly decreasing, i.e., P^^ > P*^, 
since i can send a packet to j only when Pf[t] > Pf[t]. 
In certain simple topologies, this statement can be proven 
rigorously. For a tandem network (a network whose adjacency 
matrix has non-zero elements only on the first diagonal above 
the main diagonal and the first diagonal below the main 
diagonal, see Fig. [3] for an example) with unit link capacities 
and one end-to-end flow, we have the following results: 

Lemma 1: In such a tandem network with no more than 
3 hops, the average queue-length of the DTBP algorithm is 
strictly decreasing from the source to the destination for any 
arrival rate a G (0, 1) under a Bernoulli arrival process. 

Proof: The detailed proof is provided in Appendix |B] ■ 

— © — © — Qy -O — © 

Fig. 3. A tandem network with — 1 hops. 

The method used in the proof of Lemma [T] can be extended 
to a /c-hop tandem network with a Bernoulli arrival. However 
the analysis becomes cumbersome as k increases. Thus we 
take a different approach to extend the above result to a more 
general /c-hop tandem network. 

Lemma 2: In a tandem network under our model with unit 
link capacities and one end-to-end flow with a Bernoulli arrival 
with rate a G (|, 1), the average queue-length P*^ is strictly 
decreasing from the source to the destination under DTBP. 
Proof: The detailed proof is provided in Appendix [C] ■ 

Due to the complex interactions in the dynamics of the 
queue-length in the neighboring nodes in a general network, a 
rigorous proof of a generalization of Lemma [2] remains an open 
research problem. However it can be observed from numerical 
studies of more general networks that, under the DTBP 
algorithm, the average queue-lengths are strictly decreasing 



over links with a positive net flow rate. Fig|4] shows a typical 
example in a 6 x 6 grid network. The arrows in the figure 
indicate the direction of the net flow, and the average queue- 
length of the flow in each node is shown by the number besides 
the node. 



10.1681 



12.83571 14.0564 16.376 1 17.6919 




94m 23.4056 



47021 36.8193 



14.4277 



1.9343 5.2188 8.2583 9.5971 14.356 14.4226 



Fig. 4. The net flow rate and the average queue-length in a 6 x 6 grid 
network. The average queue-lengths are strictly decreasing in the direction of 
the net flow. 



With such results and observations, we present the following 
assumption on the DTBP algorithm: 

Assumption 2: Under our system model, the DTBP algo- 
rithm converges to a optimal solution of ([T]) 
and the corresponding queue evolution P*[t] as defined in 
([5]). The optimal solutions satisfy the following property: 

If r*f > r*f then P*" > , V(i,i) e £, d e 

where r*j^ and r*f are defined as in (|7j), and P*^ and P^^ are 
defined as in 

Remark: Note that the converse of Assumption [2] is not 
necessarily true, i.e., P*^ > P*^ does not necessarily imply 



that r*j > r*f , which can be disproved by counter examples. 

Loop-Free Route Construction: Consider the network 
G = (A/", jC^c). For each commodity d, we define a subgraph 

^^ = (Ar^£^c^) as: 



AT; 

/:\{(z,j)G/::f*/ = 0}; 



= { 



if{ij)^/:^ and 
Cii otherwise. 



This is the restricted network topology that commodity d 
sees after applying the delay-aware link rate assignment. 

Then we have the following proposition: 

Proposition 2: Under Assumption |2j given a network Q = 
(A/", c) and its optimal solution fa;*, r*) given by the back- 
pressure algorithm to problem |l2| ), the subgraph defined 
as above is loop-free y d eV. 

Proof: Assume has a cycle that is composed of links 

{(n 

Cl 7 '^02)7 (^C2 5 ^03)7 ' ' ' 7 (^Cfc_i 7 '^Cfc)? (^Cfc 7 '^Cl 

By the definition of subgraph Q^, we can assume without loss 
of generality that r'^fc+i > for < i < A: - 1 and f*^^^ > 0. 
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Then by Assumption [2j we have 



p::>pz>--->p*c:>Pc. 

which is impossible. Therefore is loop-free. ■ 
Remark: As mentioned in the proposition itself, the loop- 
freeness holds only when the delay-aware link rate assignment 
is applied to the optimal solution given by DTBP. 

Example: We use the topology shown in Fig. [5ja) as an 
example to illustrate how the delay-aware link rate assignment 
works with the optimal solution given by DTBP. Each link in 
the network is assumed to have unit capacity. Node 1 is the 
source and Node 3 is the destination. 




Fig. 5. An example to illustrate the DTBP solution with delay-aware link 
rate assignment. 



Shown in Fig. |5jb) is a possible optimal solution of the 
problem, with ri2 = ^23 = 1 and r24 = = r^2 = where 
r e (0, 1]. Note that there exists a loop (2 ^ 4 ^ 5 ^ 2) in 
this optimal solution. However, analyzing the corresponding 
Markov chain shows that the DTBP algorithm can never con- 
verge to such an optimal solution. In fact, the DTBP converges 
to a solution shown in Fig. [5jc), where ri2 = r23 = 1, 
link (2,4) and (2,5) have equal and non-zero transmission 
rate in either direction, and there is no transmission on link 
(4, 5). Note that this confirms Assumption [2] in this particular 
topology, and also confirms that the optimal solutions given 
by DTBP is a subset of all optimal solutions with a special 
structure. 

If we were to apply the delay-aware link rate assignment 
to the optimal solutions shown in Fig. [5jb), it can be verified 
that the loop (2 ^ 4 ^ 5 ^ 2) still exists in the resulting 
topology. On the other hand. Fig. |5jd) shows the resulting 
topology when the delay-aware link rate assignment being 
applied to the optimal solution given by DTBP algorithm, 
which is a loop-free route. 

B. Delay-aware scheduler design 

Using the delay-aware routing of Section |IV-A| utility 
maximization can be achieved while avoiding loops, which 
leads to lower average end-to-end delays. Here, we seek to 
improve the delay performance further through delay-aware 



scheduling at each link. As argued in Section [III-B[ significant 
per hop delay improvement can be achieved by reshaping the 
service distribution through regulating the scheduler opera- 
tion. In previous works |[T6l , the notion of regulators were 
introduced to help stabilize the network with fixed single-path 
routing. Adopting a similar approach, we propose a delay- 
aware scheduler with regulated service that stabilizes (A/", £, c) 
when there exists a loop-free R[t] such that X[t] satisfies 
X e int(^A{JVj C^c)) . We assume that the arrivals take place 
after the service in each time slot, i.e., the arrivals cannot be 



served in the arriving time slot. We further assume that from 
now on all links have the same link capacity denoted by c. 



node 
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Fig. 6. Delay-Aware Scheduler Structure 

Definition 4: (Delay-aware scheduler with regulated ser- 
vice) The delay-aware scheduler consists of per-neighbor-per- 
commodity queues, i.e., the scheduler at node n maintains 
queues for each next-hop neighbor j and each commodity 
d. The servers apply regulated service discipline to shape the 
traffic. Let the service and departure of the server be S{t) and 
D(t), respectively. 

Let [t] denote the commodity d packets entering node n 
in slot t. Mathematically, 



^nM = ^n[A'^{neJ\fi 



where J^source denotes the source node set with respect to 
commodity d. Let denote the number of packets of 

commodity d arriving at queue Q'^j in slot t and Af^[t] = 
"^j^j^ Af^jlt]. The queue dynamics for each queue is given 
as: 



d: 



r,nj M J ^ ^nj 

Vn ^ d, 



(22) 



The server of the delay-aware scheduler operates with a 
token-based service discipline which consists of three compo- 
nents: 

• Arrival Splitting: 

Each commodity d packet arriving at node n goes into the 
per-neighbor-per-commodity queue Q^. with probability 



Kj[t] 



, resulting in 



E 



R'nM 



Ud 



(23) 



where R^^ [t] = j Xlt^o ^nj [^\ • ^^^^ splitting preserves 
the same mean link rates as the original R[t]. As an 
example, the arrival splitting can be done using a token- 
based scheme. 



Token Generation: 

For each link 



G £, the system maintains token 



counters m^^ for each regulator queue Q^^ as shown 
in Fig. [6] As the input to the delay-aware scheduler is 
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loop-free, the transmissions can only be unidirectional 
for certain commodity d over link (i^j). Hence m^j and 
m^^ cannot be non-zero simultaneously. Then we define 
M^ . = m"^ • 
with rate 



m^^. The token arrives at each counter 



(24) 



where 6 > 0. Note that the token arrival rates are enlarged 
by S which is allowed by the excess capacity from the 
utility maximizing solution for (A/", £, c — e). 

Service: 

In each time slot t, the token-based server chooses a 
winner commodity d*^ with a non-zero backlog for each 
link (n, j) as follows: 



<,M = <! if max, [M^M 

0, otherwise. 



c] > 0; (25) 



The server serves the commodity with the largest token 
count that exceeds the link capacity c. Note that the link 
capacity c is shared by transmissions on link (n,j) in both 
directions. Then 



if d = 

if d^d^^ 



(26) 



In case of insufficient packets in queue, the server sends 
out dummy packets to ensure a departure of c packets. 
After service, the token counter M^J^ for the commodity 
d^j is decreased by c. o 

The above algorithm does not require the knowledge of R[t] 
as long as R[t] := {Rnj[t])n,j,d is provided. 

Definition 5: (Token count process) We call M[t] a token 
count process if: 



\M[t]^u-c^^, if M[t] 



if M[t] < c*^; 



V > c 



,th 



Z[t^l] 



nth 



if M[t] 
if M[t] 



.th. 



.th 



where Z[t] is the associated token process, < u < c^'^ is the 
token generation rate and c^^ is the decision threshold. o 
Following our definitions of delay-aware scheduler, it is clear 
that the service process of each scheduler is a token count 
process. We have the following lemma showing important 
properties of the token count process, that is, given a finite 
initial state, the token count process evolves towards a bounded 
invariant region. 

Lemma 3: Let M[t] {Mi[t], . . . , Mn[t]) be the token 
count vector with token generation rate v = {ui^ . . . ^Un) and 
decision threshold c^^. Let Di[t] be the departure from token 
counter i at time t, \/l < i < n. The delay-aware scheduler 



serves M[t] with the following service discipline: 

m = 



argmax^ [Mi[t] — c^^) , 

if maxi [Mi[t] -c*^] > 0; 
0, otherwise. 



if i = z*; 
if 



The token count dynamics is as follows: 

Mi[t + 1] = Mi[t] - AM + i^i, VI < i < n. 

Let M = {M eR"" : Yl"^^-^ Mi < {n ^ l)c^^}. Suppose 
Er=i < c^^' then Vto > such that VM[to] G M^, there 
exists T < oo, such that M[to + T] e M. Moreover, M[t] e 



Proof: The detailed proof is provided in Appendix Dl 



Intuitively, Lemma [3] reveals the fact that the token count 
process keeps track of the assigned token generation rate and 
the deviation is always bounded. Lemma [3] will help us in 
proving the network stability when the delay-aware schedulers 
are used. Next we define the network capacity region for multi- 
hop networks. 

Definition 6: (Network capacity region) The network ca- 
pacity region is A(A/', c) = Co{x}, where x is the 
exogenous arrival rate vector such that x y and such that 
there exists a link rate vector r satisfying 

r'nj=0, V(n,j)^/:, 
r^^ = 0, Vn G A/-, 
= 0, Vn = d, 

j n 
d d 

Notice that the solution {X[t]^R[t]) is feasible if (x,r) 
satisfies Definition [6] such that x G A(A/', >C,c), where x = 

Based on the defined network capacity region, we now 
introduce the following proposition: 

Proposition 3: The delay-aware scheduler operating in the 
network {M^C^c) stabilizes the queues {Qnj)n,j,d, if there 
exists any loop-free solution {X[t]^R[t]) such that x G 
A(A/',/:,c-e), Ve 0. 

Proof: The detailed proof is provided in Appendix |D2| 



Remark: To stabilize the network, the proposed delay-aware 
scheduler benefits from the special properties of the token 
count process. In fact, we have proved that the delay-aware 
scheduler structure can be extended to utilize a general service 
process that satisfies certain regulation constraints and still 
achieves network stability. The existence and characterization 
of other delay-aware service processes remains an open re- 
search problem. 
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C. Delay-aware cross-layer policy 

The approaches we proposed in Section |IV-A| and |IV-B| can 
be used in the multi-layer architecture of Section |III-C[ To 
attain a utility maximization solution that reduces the expected 
end-to-end delay, we incorporate back-pressure solution to 
problem ([T]) together with delay-aware routing and delay- 
aware scheduling in the policy design. In the following policy, 
we call the queue length seen by DTBP the price, which is 
a value reflecting the queue length price instead of being an 
actual queue. 

Delay-Aware Cross-Layer Policy 

• The first layer runs DTBP policy (see Section II-B| ) 
to generate {X[t]^R[t]) for {M^C^c — e) in a virtual 
implementation using counters for prices. 

• The second layer performs the delay-aware link rate 
mapping from the first layer and generates R[t\ as in ( 17 ) 
with R[t] = R[t]. 

• The third layer uses delay-aware schedulers with regu- 
lated service in the network {M^C^c). Solution X[t] = 
X[t] is received from the first layer. Solution R[t] = R[t] 
is received from the second layer. Then the delay-aware 
scheduling is performed as described in Section |IV-B| 
which determines the actual queueing dynamics. 

This cross-layer policy is an online policy that starts to 
function while the DTBP is converging. All layers run different 
algorithms in parallel and dynamic solution updates can be 
performed among layers. 

The cross-layer policy inherits optimality (Proposition [T]) 
and loop-freeness (Proposition [2]) characteristics of the routing 
component, and the stability (Proposition [3]) of the delay- 
aware scheduling component, which lead to the following 
fundamental result: 

Theorem 1: Delay-aware cross-layer policy results in loop- 
free and stable rate assignments that arbitrarily approaches 
maximum utility solutions for a network (A/", C^c). 

V. Numerical Results 

In this section, we present numerical results for our pro- 
posed delay-aware cross-layer policy. The cross-layer policy 
is an online policy starting to function as DTBP is converging. 
All layers run different algorithms parallelly and perform 
periodical solution updates in implementation. We compare 
the delay performance of our policy with the back-pressure 
policy, which has long term optimality guarantees, and the 
more recent min-resource algorithm (see Section II-B| ), which 
has superior routing characteristics over back-pressure policy. 
As a representative topology, simulations are carried out in a 
6x6 grid network. Simulation duration is 30000 time slots. 
Moving average method is used to update R[t\ as follows: 

" to + 



to<t<to^T, 



where W is the window size and T is update period. W 
and T are both set to 5000. We assume that all links have 



unit capacity. The K parameter for the flow controller of 
the back-pressure policy is chosen to be 200. We implement 
min-resource algorithm as described in Section |II-B| with 
M = 3. For fair comparison, all queues are implemented as 
per-commodity queues. 

Two representative scenarios are considered to demonstrate 
the performance difference of the three algorithms. Scenario 1 
has parallel traffics with two commodities. Scenario 2 contains 
cross traffics where each source node maintains throughput up 
to 3 compared to 2 in Scenario 1 . The links in the center 4x4 
square are fully utilized. Scenario 2 has less excess capacity 
compared to Scenario 1. 

For better demonstration, the routing information is embed- 
ded in delay distribution graph. Since the source-destination 
pairs are symmetric, results are given for one commodity only. 
Squares and circles indicate source and destination locations 
for commodity 1 and 2, respectively, both flowing from top to 
bottom. In Scenario 1, as is shown in Fig. |7ja), the back- 
pressure policy results in loops. The delay distribution is 
heavy-tailed which also indicates that packets may experience 
looping. Fig. [TJb) demonstrates that the min-resource algo- 
rithm utilizes minimal number of links, thus attaining shortest 
paths from source to destination for each commodity. Since 
loops are avoided, the delay distribution is improved with 
the delay peaks moved towards lower values. Fig. [tJc) shows 
the network topology for delay-aware cross-layer policy. The 
selected paths are guaranteed to be loop-free. The same figure 
also shows that the delay distribution is multimodal. From 
this distribution, one observes that the majority of packets 
follow the shortest path and the others are routed over longer 
paths. The mean delay performance is improved over other 
two solutions. 

In Scenario 2, we demonstrate a situation where the network 
is heavily utilized. The existence of excess capacity is a neces- 
sary condition for back-pressure policy to form loops. Within 
a network with less excess capacity, back-pressure policy has 
less resources for loops, which leaves less chance for routing 
improvement. Fig. [SJa) shows the back-pressure result for 
such a network with increased throughput. In Fig. [8|b), we 
observe no improvement for min-resource algorithm since the 
network is already heavily loaded. Min-resource algorithm 
cannot further minimize resource usage without sacrificing 
throughput. In Fig. [SJc), we still observe significant delay 
improvements in the mean and variance of delay. The result 
suggests that the delay-aware cross-layer policy is effective in 
reducing delay regardless of network utilization. 

We further study the effect of the design parameter K used 
in the DTBP policy. It is known that the allocated flow rates 
approach the optimal rate cc* as i^T ^ oo. However, this 
also increases the equilibrium queue lengths in the DTBP and 
causes delay degradation ((13, (H, etc.) 

The delay-aware cross-layer policy offers a solution to this 
dilemma. In multi-layer architecture, the delay-aware cross- 
layer policy keeps real layer queue length separated from 
back-pressure layer queue length. Then the dependence on K 
is relaxed. Fig. [9] shows that in Scenario 1, the sensitivity 
of the delay on K is significantly reduced with delay-aware 
cross-layer policy. As expected, the delay under back-pressure 
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policy grows as K increases. In this figure, we also show 
that the end-to-end delay can be reduced by eliminating loops 
alone. However, further delay improvement can be achieved by 
utilizing the delay-aware scheduling with delay-aware routing. 



vice shaping opportunities in scheduling to reduce queue-sizes, 
while preserving the long-term optimality characteristics. 

Along with these algorithms, we also present a generic 
delay-aware design framework that can be used to develop 
other long-term optimal algorithms with favorable delay char- 
acteristics. This framework provides a unique and methodical 
approach to the design of modular solutions where end-to- 
end delay characteristics can be improved without sacrificing 
from long-term optimality. As such, our framework offers a 
promising direction for tackling the challenging problem of 
delay-aware algorithm design for wireless multi-hop networks, 
as well. To that end, in our future work, we will extend 
the insights and techniques developed in this work to the 
wireless domain, while accounting for the intrinsic challenges 
associated with the interference-limited nature of wireless 
communications . 
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Appendix 



A. Proof of Proposition [7] 



Let the optimal solution to (12) be (ic*,r*), we define 



X 



X 



(27) 



f * as defined in Equations ([18]) and ([19). 



Since x* — a?*, the optimal value of the objective function 
(12) is not affected. Thus, to show that (aj*,f*) is also an 
optimal solution to ( [T2| ), it is sufficient to show that the pair 
(a^*,^*) satisfies the constraints (13)-(16). 



• Verifying Equation ( 13 ) and ( 14 ): 

*^ > because xf > for all s,deV\ r*f > for all 



X 



{i^j) G C and d G P by its definition in Equations (18) and 



Verifying Equation ( 15 ): 



By the definition of and the fact that r*j^ > 0, V(z, j) G 
£, G P, we have f*/ < r*/, V(z, j) e C.W e V. Then 



Verifying Equation ( [T6| ): 



Noticing that for any real number Z, (Z)+ = 
Equations (18) and ( [T9| ) can be rewritten as: 

l{{r:f-r*f) + \r*f-r*t\); 



\Z\), 



^3 



^d 



{{r*f-r*f) + \r*t-rlf\). 



Subtracting the above two equations, we have 



jl 



*J 3 

Thus, the following equation holds: 

/ _j ' mi 

{m,i) 



(28) 



{i.j)ec 



(a) 



ji ) 



^*d 



' 3^ ) 



<o, 



where (a) follows from ( [28] ) and (b) follows from ( [27]). 
Therefore, (x*,^*) is also an optimal solution to \\2\. 

B. Proof of Lemma \I\ 

Lemma [T] can be proven by constructing the queue-evolution 
Markov chain under the DTBP algorithm and studying the 
steady- state behavior of the queue-length evolution from zero 
state. 

To simplify the notation, we denote the nodes as node 
to node n in a n-hop tandem network, where node is the 
destination, and node n is the source. Since only one flow 
is assumed to exist, we omit the superscript d indicating the 
destination of the flow. 

Case 1: n = 1. In this case, node 1 receives a packet 
with probability a in each time-slot. Whenever node 1 has one 
packet, it sends the packet to the destination. Thus Prob{Pi = 
1} = a, and Prob{Pi = 0} = 1 - a, thus 

P * = a > = Pq* 

when a G (0, 1). 

Case 2: n = 2. For brevity, we denote the state of the 
Markov chain (P2,Pi) as in the Table |lj Since the Markov 

TABLE I 

The states in the Markov chain of a 2-hop tandem network 





State 




State 


(0,0) 


1 


(1,1) 


4 


(0,1) 


2 


(2,0) 


5 


(1,0) 


3 


(2, 1) 


6 



chain itself is messy to plot, we only present its state transition 
matrix 

b a ' ' ' ' 

b • a • 

b • a • • 

b • a • 

b • a 

b • a 



A = 



11 



where b = 1 — a, and we use "•" to represent for clarity. 
Solve the equation 

AtT = TT 

and the probability constraint 



to get the steady-state distribution of this Markov chain, and 
the average queue-length can be calculated by 

Pi = 7r2 + 7r4 + 7r6, 

P2 = TTs + 7r4 + 2(7r5 + TTe). 

It can be verified that 

P2* > Pi > PS =0 



when a G (0, 1). Shown in Fig. 11 is the average queue-lengths 
as a function of a in a 2-hop tandem network. 
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Fig. 10. The average queue-length in a 2-hop tandem network under 
Bernoulli(a) arrival. The queue-lengths are strictly decreasing from source 
to destination. 



Case 3: n = 3. For brevity, we denote the state of 
the Markov chain (P3,P2,Pi) as in the Table With this 



TABLE II 

The states in the Markov chain of a 3 -hop tandem network 



(P3,P2,Pl) 


State 


(P3,P2,Pl) 


State 


(0,0,0) 


1 


(1,0,1) 


9 


(1,0,0) 


2 


(2,0,1) 


10 


(0,1,0) 


3 


(3,0,1) 


11 


(1,1,0) 


4 


(2,1,1) 


12 


(0,2,0) 


5 


(3,1,1) 


13 


(1,2,0) 


6 


(2,2,1) 


14 


(2,2,0) 


7 


(3,2,1) 


15 


(3,2,0) 


8 







numbering of the states, the state transition matrix is given by 

A = 



b a 



b a 



b a 
b a 
b a 
b a 



b a ■ ■ 

b a 

b a 
b a 

b a 



b a 



b a 



where b = 1 — a, and we use "•" to represent for clarity. 
Solve the equation 

AtT = TT 

and the probability constraint 

i 

to get the steady-state distribution of this Markov chain, and 
the average queue-length can be calculated by 

15 



P2 = TTs + 7r4 + 7ri2 + 7ri3 

+2(7r5 + TTe + TTt + TTg + 7ri4 + TTis), 
P3 = 7r2 + 7r4 + TTe + TTg 

+2(7r7 + TTio + 7ri2 + TTm) 

+3(7r8 + 7rii +7ri3 +7ri5). 
It can be verified analytically or numerically that 

P3 > P2* > Pi > ^0 = 



when a G (0, 1). Fig. 11 shows the average queue-lengths as 
a function of a in a 3 -hop tandem network. 

Remark: As we can see from the above analysis, the 
dimension and the number of states in the Markov chain 
grow fast as n increases. It is cumbersome and impractical to 
precisely analysis the Markov chain to prove the generalization 
of Lemma [T] 

C. Proof of Lemma^ 

To simplify the notation, we denote the nodes as node 
to node n in a n-hop tandem network, where node is the 
destination, and node n is the source. Since only one flow 
is assumed to exist, we omit the superscript d indicating the 
destination of the flow. 

Let e be defined as e = 1 — a. By the feasibility constraint 
( p?) and the flow conservation constraint ([16]), we have 

rl^-l^rU,^ < 1 

> 1-e 



12 
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Fig. 11. The average queue-length in a 3 -hop tandem network under 
Bernoulli(a) arrival. The queue-lengths are strictly decreasing from source 
to destination. 



for alH = 1, 2, • • • , n. Thus we have Vi-i^i < e/2 for all i. 

Also, note that under DTBP algorithm, \Pi[t] -Pi-i[t]\ < 3 
for all i = 1, 2, • • • , n — 1 and any t. This is because each 
node i can get at most 2 packets from its neighbors (2 is the 
degree of all intermediate nodes in a tandem network), while 
its neighboring nodes can send out at most 1 more packet to 
other nodes. To receive one packet from both of its neighbor, 
the queue length at node i is at least one less than that of its 
neighbor. This implies that the steady state distribution satisfies 



^(P)1||P,_P,_,I>3}(P)=0, Vi = l,2,- 



1, 



where 1|^}(P) is the indicator function of a set A and 
1|^}(P) = 1 if P G ^ and otherwise. Note that 
Pi-i[t] — Pi[t] < 3 also holds for the source node n. 
Thus, we have 

p* - p*_, 

= ^7r(P)(Pi-Pi_i) 

P 

= ^7r(P)l{p,>p._,}(P)(Pi-Pi_i) 

P 

-Y,<p)Up.<p.-.}{p){Pi-i-Pi) 

p 

>(,) ^7r(P)l{p,>p._,}(P) 

P 

-3^7r(P)l{p,<p._j(P) 

P 

=(6) - 3r*_i^^ 

> l-6-2r*_i,, 

> 1 -2e 

> Oife<-, 

where (a) holds because Pi — Pi-i > 1 on the set {P^ > Pi-i} 
and Pi-i — Pi < 3 from the discussion above, and (b) holds 
from the definition of the DTBP algorithm. This argument 
holds for alH = 1, 2, • • • , n, thus the lemma is proven. 

D. Proof of Proposition |j] 



1 ) Proof of Lemma |jf First we show that M is attractive 
and that the process M[t] evolves towards M. Let t be such 
that M[t] ^ M. Then YTi^i^AA > + ^V^- Then 



nth 



- > c*^. So there exists j such that 
Mj[t] > c^^, which implies J^^^^ AM = c^^. Then we have 



]M,[t] 



i=l 



i=l 



Let T = [2%^^^^^^^^! . Then M[t^T]eM, which 
proves Vt > 0, VM[t] ^ there exists T < oo such that 
M[t + T] e M. 

Next, we prove that M. is invariant in M. Define M' 



[M e 



nc 



.th 



< ElLi^* < (n + l)c^^}. We first 



prove that any M[t] G M\M\ M[t ^ 1] e M, i.e., M[t] 
cannot escape M\M' without being in M'. Then we prove 
any M[t] G M\ M[t + 1] G M, which shows that any 
solution in cannot leave M. 

To see the first claim holds, for any M[t] G M\M\ 

Zti < nc'\ Then ELi + 1] < ELi + 

5:^^! < (n + l)c'^. So M[t + 1] G M. 

Now we prove any M[t] G M\ M[t+1] G A^. Let M[t] G 
A^'. Then ^^iLi ^4^1 ^ there exists j such that 

M^[t] > c^^. Thus E^^i AM = follows. Together with 
ELi ^4^] < + we have 



< (n + l)c^ 



So M[t + 1] G Hence is invariant in M. And the 
lemma follows. 

2) Proof of Proposition^ As described in section [IV-B| we 
assume that packets are sent from sources to destinations along 
known multi-paths. We assume that the exogenous arrival rates 
and the steady state link rates are available from {X[t]^ R[t]), 
the loop-free solution such that x G A(A/', £, c — e). As X^[t] 
is the exogenous arrival of commodity d at source node n, we 
assume IE[(X^M)2] < oo. 

We acquire the arrival splitting ratio from the back-pressure 



algorithm in steady state, i.e., the definition in (|23])(|24|) be- 
comes 



E 



(29) 



5. (30) 



Let Hjnax < oo be the maximum path length since the paths 
are loop-free. Xld^nj made strictly less than c by 

guaran- 



choosing sufficiently small 5, i.e., 5 < |^ ^ ||^| 
tees that the link capacity constraint is satisfied. We assume 
M[0] e {M : EdevM^j < (1^1 + V(n,i) G £}. 
We define the Lyapunov function as 
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where Qf^j [t] is the queue for commodity j transmissions from 
node n to node j at time t. 

For any T > .^^^^^ ^e consider the T step drift of the 
Lyapunov function candidate, 



AV{Q[t]) 



(l^l+i) 
anc 

V{Q[t + T])-V{Q[t]) Q[t],M[t] 



From (l29l), we have 



■t+T-l 



E 



The following analysis before ( |32| ) is similar to the analysis 
in [il4il . The queueing dynamics over time T satisfies 



E 



(31) 

where Ai[t] = X^[t]l[^^^,_^j + EfeeAT^fcnW- Squaring 
both sides of ( [3T] i and moving terms to the left side, we have 

{QUi + T]f-{QUi]f 

t-\-T-l t+T-1 t+T-1 

<( E <-M)' + ( E <-m)'-2q:^,m e 



r=t >- Z-^meN nm 

t+T-1 r 1 



E ^ 

t+T-1 



E E^ 



DL[r] M[t\ 



TDd 



(33) 



We know that ^n, < 1 and Zdev ^4 M < + 
By Lemma we have T^dev ^njl^] < (1^1 + Vt > 0. 



r=t 

t+T-1 



t+T-1 



+ 2 ^ <-[r](Qt.M- E <-M)^ 

r=t r=t 
t+T-1 t+T-1 

<( E <-M)' + ( E <-M)'-2Qt-W E 



t+T-1 



For each D^[t], we have 



t+T-1 t+T-1 



T = t 

t + T-1 



r=t 



r=t 



r=t 



2 ^ <-[r]Q:^,.M 



r=t 
t+T-1 



t+T-1 



Moving terms to the left side and taking the absolute value of 
both sides, 



=( E <-M) + ( E 



t+T-1 



t+T-1 



+ 2Qt-w( E <-M- E 

r=t r=t 

Then the T step drift of the Lyapunov function satisfies 

Avm]) 



t+T-1 



t+T-1 



r=t 



V{Q[t + T])-V{Q[t]) Q[t],M[t] 



^ d n j 

2 



E E 

<{\V\ + l)c. 



M4[t + r-i]-M4M 



-EEE(Q"iW) Q{tiM[t] 

d n j 

r t+T-1 t+T-1 

^EEE^ ( E <-M)' + ( E <-M)'|^w 

d n j r=t r=t 

+2E[EEEQn.w 

t+T-1 t+T-1 

^^+2EEEQiM 



As a result of sending out dummy packets in case of in- 
sufficient packets in Qf^j[t], the departure process D!^j[t] is 
deterministic given its associate token counts before time t. 
So we have 



t+T-1 t+T-1 

e[J2 Di^[r]\M[t]]= J2 



T=t 



T = t 



Therefore 



■E 



t+T-1 



t+T-1 



J2 <-M- E 



(32) 



t+T-1 

r=t 

<(|D| + l)c. 



t+T-1 



[ J2 Dt^[r]\M[t]\- J2 



(34) 
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Substituting ( [33] ) into ( [32] ), we have 
AV{Q[t]) 



d n j 



rt + T-1 



t+T-1 



-fl 



Y^[Dtn[r]\M[t] 

T=t keAf 

t+T-l 

- J2 ^[Dij[r]\M[t] 

d n j 
t+T-1 _ _ 



TDd 



mGA/" nm 



fT-l 

E E^^ 



TDd 



r=t keM 

t+T-l 

. E Efe-H M[t\ 



(35) 



Applying Q and (|30]( to ([35]l , then 
(l35l 



^^+2EEEQtM 

d n j 



t+T-l f>d 

+ E E^^^nM y- "V 

t+T-l >^ 

- ( E - + 
<s+2EEEQn.w{-^'^+ (1^1 + 1)4- 

d n j ^ ^ 

Hence for any T > ^^^il^^ the T step drift satisfies 

AV{Q[t]) < B-2r,5'EEEQ"iW' (36) 

d n j 

where S' = TS - {\V\ ^ l)c> 0. Applying Lemma 2 in |[T4ll 
to ([36]), the proposition is proved. 
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